Function Tools¶

There are some additional tools that are available within a runner.

The first of these is loaded, which is available for dependencies. The second of which is manifest, which allows access to the manifest file that’s used to track runner states.

loaded¶

Loaded was covered in the dependency tutorials, but as a quick refresher:

Whenever you create two functions that are intended to be linked together, you can access the parent’s output from within the children using the loaded object. Basically, loaded is as if you had called the function right there.

This means that if your parent function returns more than one item, you should be careful to collect the correct one in the child. Lets say we have a function that returns a result, and a status:

[1]:

from remotemanager import Dataset

def f1(x):
    return x, True

def f2(y):
    return loaded[0] + y

[2]:

ds1 = Dataset(f1, name="a", skip=False, local_dir="temp_dep_local", remote_dir="temp_dep_remote")
ds2 = Dataset(f2, name="b", skip=False, local_dir="temp_dep_local", remote_dir="temp_dep_remote")

ds1.set_downstream(ds2)

ds2.append_run({"x": 1, "y": 10})

ds2.run()

ds2.wait(1, 5)

ds2.fetch_results()

ds2.results

appended run runner-0
appended run runner-0
Staging Dependency
  [0] dataset-a-526bebb1... Done, 1/1 Runners staged
  [1] dataset-b-233f50b5... Done, 1/1 Runners staged
Done
Transferring for 2 Runners
Transferring 7 Files... Done
Remotely executing 1 Runner
Fetching results
Transferring 2 Files... Done

[2]:

[11]

manifest¶

The manifest is a file that allows the runners to store their status, for later collection. As it acts somewhat like a logfile, you also have the ability to log to it.

This is done by accessing the write method:

[3]:

def function_with_log(inp):
    manifest.write(f"function was called with input: {inp}")

[4]:

ds = Dataset(function_with_log, skip=False)

ds.append_run({"inp": True})

ds.run()

ds.wait(1, 10)

ds.fetch_results()

ds.results

appended run runner-0
Staging Dataset... Staged 1/1 Runners
Transferring for 1/1 Runners
Transferring 5 Files... Done
Remotely executing 1/1 Runners
Fetching results
Transferring 2 Files... Done

[4]:

[None]

Here, our function returns nothing, but we can still access what was logged at the history attribute of the runner:

[5]:

for time, log in ds.runners[0].history.items():
    print(time, log)

2025-04-04 12:56:57/0 created
2025-04-04 12:56:57/1 staged
2025-04-04 12:56:57/2 transferred
2025-04-04 12:56:57/3 submit pending
2025-04-04 12:56:57/4 submitted
2025-04-04 12:56:57/5 started
2025-04-04 12:56:57/6 function was called with input: True
2025-04-04 12:56:57/7 completed
2025-04-04 12:56:58/0 satisfied

Any newlines will also be respected:

[6]:

ds.append_run({"inp": "\nthis demonstrates a newline"})

ds.run()

ds.wait(1, 10)

ds.fetch_results()

ds.results

appended run runner-1
Staging Dataset... Staged 1/2 Runners
Transferring for 1/2 Runners
Transferring 5 Files... Done
Remotely executing 1/2 Runners
Fetching results
Transferring 2 Files... Done

[6]:

[None, None]

[7]:

for time, log in ds.runners[1].history.items():
    print(time, log)

2025-04-04 12:56:58/0 created
2025-04-04 12:56:58/1 staged
2025-04-04 12:56:58/2 transferred
2025-04-04 12:56:58/3 submit pending
2025-04-04 12:56:58/4 submitted
2025-04-04 12:56:58/5 started
2025-04-04 12:56:58/6 function was called with input:
this demonstrates a newline
2025-04-04 12:56:58/7 completed
2025-04-04 12:56:59/0 satisfied